The present invention relates a method and an apparatus for video decoding capable of reducing an adverse effect on decoded video by a delay in a stream, which is caused by congestion of a transmission line or retransmission of the stream as a countermeasure against a transmission path error, when transmitting a coded stream obtained by coding video data, through the communication line. Further, the invention relates to a program storage medium containing a video decoding program for implementing the video decoding method by software.
In recent years, we have greeted the age of multimedia in which representation media such as audio, video, and the like are integrally handled, and the conventional information media, i.e., means for transmitting information to men, such as newspapers, magazines, televisions, radios, and telephones, have been adopted as the targets of multimedia. Generally, xe2x80x9cmultimediaxe2x80x9d means media representing not only characters but also diagrams, speeches, and especially pictures simultaneously in association with each other. In order to handle the conventional information media as the targets of multimedia, it is necessary to express the information in digital format.
However, when the quantity of data possessed by each of the above-described information media is estimated as the quantity of digital data, audio having quality for telecommunication requires 64 kbits per second while the quantity of data per character is only 1-2 byte. Further, video having quality for present television broadcasting requires more than 100 Mbits per second. So, with respect to the information media such as a telephone and a television, it is not practical to handle such massive data as it is in digital format. Although visual telephones have already been put to practical use by ISDN (Integrated Services Digital Network) having a transmission rate of 64 kbps-1.5 Mbps, it is impossible to transmit video data obtained by a television camera as digital data through ISDN.
So, a data compression technique is demanded. For example, for a visual telephone, a video compression technique such as H.261 or H.263 standard, which have been standardized by ITU-T (International Telecommunication Unionxe2x80x94Telecommunication Sector), is employed. Further, according to a data compression technique of MPEG1 standard, it is possible to store video data as well as audio data in an ordinary music CD (compact disk).
MPEG (Moving Picture Experts Group) is an international standard of video data compression, and MPEG1 is a standard for compressing video data to 1.5 Mbps, i.e., data of a television signal to about 1/100. Since the transmission rate of the targets to which MPEG1 is directed is limited to about 1.5 Mbps, MPEG2, which compresses video data to 2xcx9c15 Mbps, has been standardized to meet the demands for higher picture quality.
Furthermore, under the existing circumstances, standardization of MPEG4 is now proceeded by the working group for standardization of MPEG1 and MPEG2 (ISO/IEC JTC1/SC29/WG11), and this MPEG4 enables coding and processing of picture data in object units to realize new functions required in the age of multimedia. At the beginning, MPEG4 has aimed at standardization of low-bit-rate coding, but the target of standardization is now extended to more versatile high-bit-rate coding adaptable to an interlace picture.
By the way, a feature of MPEG4 resides in that picture data corresponding to plural picture sequences (objects) are simultaneously coded and transmitted. This scheme enables constitution of one moving picture composed of plural objects. In this scheme, for example, a foreground and a background are handled as different picture sequences (different objects), and the frame frequency, picture quality, bit rate, etc. are separately changed for each picture sequence. Further, when plural picture sequences are arranged in the horizontal or vertical direction on a display area like multi-window, the user can extract or enlarge only his/her favorite picture sequence.
In order to code an object having no shape such as a background, generally employed is a coding method in which only pixel value signals indicating pixel values of the object are coded as picture data, such as MPEG2. On thy other hand, in order to code an object having a shape such as a foreground, employed is a coding method in which not only pixel value signals indicating pixel values of the object but also shape signals indicating the shape of the object are coded as picture data.
Further, in MPEG2 and MPEG4, each picture sequence is coded for each of pictures constituting the picture sequence (it corresponds to a frame in MPEG2, and a VOP in MPEG4), and some of the pictures are subjected to a differential coding process (inter-frame predictive coding) in which difference data between picture data of a picture to be encoded (target picture) and predictive data obtained from picture data of a picture for which coding has already been completed (already-processed picture) are encoded.
However, with respect to an object having a shape, its shape and size vary with time. So, in the coding process based on MPEG4, the coding algorithm for a shape signal and a pixel value signal greatly depends on the shape of the object in the already-processed picture. Therefore, the coding process based on MPEG4 is less resistant to a transmission error of a coded stream and has more difficulty in picture recovery utilizing inter-picture correlation or picture correction by picture processing, as compared with the coding process based on MPEG2 which handles picture data of a picture whose shape and size do not vary. In other words, in a decoding system based on MPEG4, when a transmission error occurs in a coded stream, the quality of decoded picture is considerably degraded.
In MPEG4, as described above, one moving picture is composed of plural picture sequences (objects), and each picture in one moving picture is a frame while each picture in one picture sequence is a VOP (Video Object Plane). When one moving picture is composed of one picture sequence, a frame is equal to a VOP.
Hereinafter, a decoding system adaptable to MPEG4 will be described specifically.
Generally, in a data transmission path (communication line), transmission delay of data occurs due to retransmission of data or the like which is a countermeasure against noise and congestion. Therefore, in a coded stream outputted from a video coding apparatus for coding picture data to output a coded stream or from a video transmission apparatus for transmitting a coded stream, the order of bit streams constituting the coded stream (the order of bit streams at transmission) is different from that when received at the receiving end (the order of bit streams at reception). To be specific, the order of bit streams corresponding to some VOPs in the coded stream at transmission is different from that at reception.
As a countermeasure against this problem, when a coded stream is divided into transmission units called xe2x80x9cpacketsxe2x80x9d at the transmitting end, each packet is given a number indicating its turn in the transmission order. Thereby, at the receiving end, VOP data in the received coded stream can be rearranged such that its order becomes equal to that at transmission, on the basis of the number given to each packet.
FIG. 30 is a block diagram for explaining a conventional video decoding apparatus 1000. The video decoding apparatus 1000 receives coded streams Vin corresponding to plural objects constituting one moving picture, and rearranges VOP data in the coded stream corresponding to each object. Thereafter, the video decoding apparatus 1000 decodes the coded streams Vin, and composites decoded picture data corresponding to the respective objects. Further, at the transmitting end, each VOP data in the coded stream Vin corresponding to each object is given information indicating the time at which the VOP data is to be displayed at the receiving end (VOP display time).
The video decoding apparatus 1000 includes a rearrangement unit 1001 and a decoding unit 1002. The rearrangement unit 1001 receives a coded stream Vin corresponding to a specific object from the transmitting end, rearranges streams (VOP data) corresponding to the respective VOPs in the coded stream such that the order of these VOP data is equal to the order of the VOP data at the transmitting end, and then outputs the coded stream Vin after the rearrangement as a video stream VOPstr. The decoding unit 1002 decodes the video stream VOPstr with reference to reference picture data VOPref as necessary to generate decoded picture data VOPout. Further, the video decoding apparatus 1000 includes a video memory 1003 which stores the decoded picture data VOPout from the decoding unit 1002, and supplies picture data required for the above-mentioned decoding process, amongst the decoded picture data VOPout stored, as reference picture data VOPref to the decoding unit 1002.
The rearrangement unit 1001 is provided with a buffer memory 1101, and a display time output unit 1102. The buffer memory 1101 temporarily stores the coded stream Vin supplied from the transmitting end. The display time output unit 1102 includes a timer, and generates a display time (specified display time) Td at which each of frames constituting the moving picture is to be displayed, which display time is specified at the receiving end. The rearrangement unit 1001 outputs, as a video stream VOPstr, specific VOP data from the buffer memory 1101 to the decoding unit 1002 on the basis of the specified display time Td. The specific VOP data is VOP data stored as the coded stream Vin in the buffer memory 1101, whose VOP display time is later than the VOP display time of VOP data which has already been outputted as a video stream VOPstr to the decoding unit, and earlier than the specified display time Td. The decoding unit 1002 includes a decoder 1103 for subjecting VOP data which has been coded by intra-frame coding to a decoding process without reference to the reference picture data VOPref, and subjecting VOP data which has been coded by inter-frame predictive coding to a decoding process with reference to the reference picture data VOPref, thereby generating decoded picture data VOPout of each of VOPs constituting the corresponding object.
In FIG. 30, although the rearrangement unit 1001, the decoding unit 1002, and the video memory 1003 correspond to one object, the video decoding apparatus 1000 actually includes rearrangement units 1001, decoding units 1002, and video memories 1003 as many as, at least, the maximum number of objects which are components of the moving picture to be displayed simultaneously. The maximum number of objects to be displayed simultaneously is equal to or smaller than the number of all objects which appear in one moving picture. Further, each of the rearrangement units in the video decoding apparatus does not necessarily have a display time output unit 1102, and a rearrangement unit corresponding to at least one object may have a display time output unit 1102. In this case, to the buffer memory 1101 of the rearrangement unit having no display time output unit 1102, a specified display time Td is supplied from the rearrangement unit having a display time output unit 1102.
Further, the video decoding apparatus 1000 is provided with a video compositor unit 1004 and a display unit 1005. The video compositor unit 1004 outputs, on the basis of the specified display time Td, a display picture selecting signal VOPsel for selecting decoded picture data corresponding to a VOP whose VOP display time is nearest to the specified display time Td, to the video memory corresponding to each object, and reads the decoded picture data corresponding to a VOP whose VOP display time is nearest to the specified display time Td, as display picture data VOPdes, from the video memory corresponding to each object, and composites the display picture data read from the video memories corresponding to the respective VOPs to make composite picture data Vout. The display unit 1005 displays the picture corresponding to one scene of the moving picture, on the basis of the composite picture data Vout. The display unit 1005 displays the moving picture by continuously displaying the pictures corresponding to the respective scenes of the moving picture, at the specified display times Td.
Hereinafter, the operation of the conventional video decoding apparatus 1000 will be described.
When coded streams Vin corresponding to plural objects constituting one moving picture are inputted to the video decoding apparatus 1000, the coded stream Vin corresponding to each object is stored in the buffer memory of the rearrangement unit corresponding to the object.
Since rearrangement of bit streams in a coded stream, decoding of the coded stream, and storage of decoded picture data into the video memory are common to the respective objects, these processes will be described for one object.
In the buffer memory 1101, all of the inputted coded streams are stored until the quantity of data reaches the capacity of the buffer memory 1101. Thereafter, every time a coded stream is inputted, the coded streams stored in the buffer memory 1101 are successively deleted in the order in which the streams were inputted to the buffer memory 1101, by a quantity of data equivalent to that the inputted coded stream.
In the display time output unit 1102 having a timer, a display time (specified display time) Td of each of pictures constituting the moving picture at the receiving end is generated, and information indicating the specified display time Td is output to the buffer memory 1101. In the buffer memory 1101, specific VOP data is selected from the buffer memory 1101 on the basis of the specified display time Td, and the selected VOP data is output as a video stream VOPstr to the decoding unit 1002.
The specific VOP data selected at this time is VOP data whose VOP display time is earlier than the specified display time Td and later than the VOP display time of the last VOP data amongst the VOP data which have already been outputted to the decoding unit 1002 at this point of time. Thereby, in the rearrangement unit 1001, rearrangement of VOP data is carried out so that the order of VOP data in the received coded stream (hereinafter also referred to as xe2x80x9creceiving-end VOP orderxe2x80x9d) becomes equal to the order of VOP data in the coded stream outputted from the transmitting end (hereinafter also referred to as xe2x80x9ctransmitting-end VOP orderxe2x80x9d). To be specific, even if the transmitting-end VOP order is difference from the receiving-end VOP order due to congestion of the communication line and thereby the VOP data are input to the buffer memory 1101 in the order different from the transmitting-end VOP order, the VOP data are output from the buffer memory 1101 in the correct VOP order equal to the transmitting-end VOP order, as a video stream VOPstr, to the decoding unit 1002.
In the decoding unit 1002, the inputted video stream VOPstr is decoded by the decoder 1103, and decoded picture data VOPout of each VOP is output as already-decoded data. The decoded picture data VOPout outputted from the decoding unit 1002 is temporarily stored in the video memory 1003. When the inputted video stream VOPstr is VOP data corresponding to a VOP which has been subjected to intra-VOP coding, the decoding unit 1002 subjects the video stream VOPstr to intra-VOP decoding without referring to the already-decoded data stored in the video memory 1003. On the other hand, when the inputted video stream VOPstr is VOP data corresponding to a VOP which has been subjected to inter-VOP coding, the decoding unit 1002 subjects the video stream VOPstr to inter-VOP predictive decoding with reference to the already-decoded data (reference picture data) VOPref stored in the video memory 1003.
Further, the information indicating the specified display time Td from the display time output unit 1006 is also supplied to the video compositor unit 1004. In the video compositor unit 1004, a display picture selecting signal VOPsel for selecting decoded picture data of a VOP whose VOP display time is nearest to the specified display time Td is output to the video memory 1003 on the basis of the specified display time Td. Then, decoded picture data of a VOP whose VOP display time is nearest to the specified display time Td is output from the video memory 1003, as display picture data VOPdsp, to the video compositor unit 1004. At this time, plural pieces of display picture data VOPdsp corresponding to the respective objects are supplied from the video memories corresponding to the respective objects to the video compositor unit 1004.
When the decoded picture data stored in the video memory 1003 has already been output to the video compositor unit 1004 at the time when the display picture selecting signal VOPsel from the video compositor unit 1004 is input to the video memory 1003, or when no decoded picture data is stored in the video memory 1003, no decoded picture data is output from the video compositor unit 1004 in response to the display picture selecting signal VOPsel.
In the video compositor unit 1004, the display picture data VOPdsp corresponding to the respective objects are composited, whereby composite picture data Vout is output to the display unit 1005. This composite picture data Vout corresponds to one scene of the moving picture. In the display unit 1005, the picture corresponding to each scene of the moving picture is displayed on the basis of the composite picture data Vout. Further, the display unit 1005 displays the moving picture by continuously displaying the pictures of the respective scenes, on the basis of the specified display times Td.
FIGS. 31(a) and 31(b) are diagrams for explaining the flow of processing on a coded stream in the conventional video decoding apparatus. In these figures, a coded stream corresponding to one picture sequence (object) is processed.
Initially, a description will be given of the case where the order of VOP data in a received coded stream is equal to the order of VOPs at the transmitting end, with reference to FIG. 31(a).
When there is no defect in the coded stream due to transmission path delay, for example, VOP data of VOP1xcx9cVOP4 are input, as a coded stream Vin, to the video decoding apparatus 1000 in the order of VOP display times T1xcx9cT4 (T1 less than T2 less than T3 less than T4) which are set for the respective VOPs.
Then, these VOP data are input to the buffer memory 1101 of the rearrangement unit 1001 to be stored in it for a predetermined period of time (delay time Bdt1). When the predetermined period of time has passed, the VOP data of VOP1xcx9cVOP4 are output to the decoding unit 1002, as video streams VOPstr1xcx9cVOPstr4, on the basis of the specified display time Td from the display time output unit 1102. The delay time Bdt1 of the VOP data in the buffer memory 1101 is determined according to the capacity of the buffer memory 1101. The larger the capacity of the buffer memory 1101 is, the longer the delay time is. In this case, the buffer memory 1101 has enough capacity for VOP data of two VOPS.
In the decoding unit 1002, the video streams VOPstr1xcx9cVOPstr4 of the respective VOPs are decoded in the order of the VOP display times T1xcx9cT4. The decoded picture data of the respective VOPs are temporarily stored in the video memory 1003 and, thereafter, outputted to the video compositor unit 1004, as display picture data VOPdsp1xcx9cVOPdsp4, according to the display picture selecting signal VOPsel from the video compositor unit 1004.
Next, a description will be given of the case where the order of VOP data in a received coded stream is not equal to the order of VOPs at the transmitting end, with reference to FIG. 31(b).
When the order of VOP data in the codes stream is not correct, VOP data of VOP1xcx9cVOP4 are input to the rearrangement unit 1001 of the video decoding apparatus 1000, as a coded stream Vin, in the order in which the VOP data arrived at the apparatus 1000, i.e., in the order of VOP1, VOP3, VOP2, and VOP4.
Then, these VOP data are input to the buffer memory 1101 of the rearrangement unit 1001 to be stored in it for a predetermined period of time (delay time Bdt2 ). When the predetermined period of time has passed, the VOP data of VOP1xcx9cVOP4 are output as video stream VOPstr1xcx9cVOPstr4 to the decoding unit 1002, on the basis of the specified display time Td from the display time output unit 1102. At the point of time when the VOP data of VOP2 is output from the rearrangement unit 1001 on the basis of the specified display time Td, the VOP data of the VOP2 is stored in the buffer memory 1101 and, therefore, the VOP data of VOP2 which has arrived at the video decoding apparatus 1000 after the VOP data of VOP3 is output to the decoding unit 1002 before the VOP data of VOP3. The buffer memory 1101 has enough capacity for VOP data of three VOPs.
In the decoding unit 1002, the video streams VOPstr1xcx9cVOPstr4 of the respective VOPs are decoded in the order of the VOP display times T1xcx9cT4. The decoded picture data of the respective VOPs are temporarily stored in the video memory 1003 and, thereafter, outputted to the video compositor unit 1004, as display picture data VOPdsp1xcx9cVOPdsp4, according to the display picture selecting signal VOPsel from the video compositor unit 1004.
However, the quantity of VOP data which can be stored in the buffer memory 1101 of the rearrangement unit 1001 is limited. Therefore, although there arises no problem hen the degree of transmission path delay is so small that the rearrangement unit 1001 can absorb the disorder of arrival VOP data due to the transmission path delay, if there occurs a large transmission path delay which disables the rearrangement unit 1001 from absorbing the disorder of arrival VOP data, this delay adversely affects the decoding operation.
FIG. 32 is a diagram for explaining the flow of processing performed on a coded stream in the conventional video decoding apparatus. FIG. 20 shows the case where VOP data of VOP2 arrives at the video decoding apparatus with a delay due to which the VOP data of VOP2 cannot be rearranged, with respect to VOP data of other VOPs in the rearrangement unit 1001. In other words, the arrival time of the VOP data of VOP2 is delayed more than the delay time in the buffer memory 1101.
When VOP data is input to the rearrangement unit 1001 at a correct input timing, it is output from the rearrangement unit 1001 to the decoding unit 1002 at an output timing after the above-mentioned buffer delay time has passed from the correct input timing. Therefore, even when the VOP data arrives at the video decoding apparatus 1000 with a delay, if the delayed VOP data is input to the rearrangement unit 1001 before the output timing, it is normally output to the decoding unit 1002.
However, when the transmission delay of VOP data of VOP2 is considerably large as shown in FIG. 32, there arises a situation where the VOP data of VOP2 is not yet stored in the buffer memory 1101 of the rearrangement unit 1001 at the time when the VOP data of VOP2 should be output to the decoding unit 1002 (output timing of VOP2). In this situation, the VOP data of VOP2 is not decoded by the decoding unit 1002.
Thereafter, at the time when VOP data of VOP3 is output to the decoding unit 1002 (output timing of VOP3), the VOP data of VOP3 stored in the buffer memory 1101 of the rearrangement unit 1001 is output to the decoding unit 1002. In the decoding unit 1002, the VOP data of VOP3 is decoded after decoding VOP data of VOP1.
Although the VOP data of VOP2 arrives at the video decoding unit 1000 at the time when VOP data of VOP4 is output to the decoding unit 1002 (output timing of VOP4), the VOP data of VOP2 is discarded, and only the VOP data of VOP4 is output to the decoding unit 1002. This is because the VOP data of VOP2, which is earlier in decoding order than the VOP data of VOP3, cannot be decoded after decoding the VOP data of VOP3. In the decoding unit 1002, the VOP data of VOP4 is decoded after decoding the VOP data of VOP3.
On the other hand, decoded picture data of VOP1, VOP3, and VOP4 supplied from the decoding unit 1002 art stored in the video memory 1003, and the decoded picture data of each VOP is output, as display picture data VOPdsp, from the video memory 1003 to the video compositor unit 1004, at the time to display each VOP. However, since the VOP data of VOP2 (video stream VOPstr2) is not input to the decoding unit 1002, decoded picture data of VOP1 (display picture data VOPdsp1) is output at the time to display VOP2, as decoded picture data of VOP2, from the video memory 1003 to the video compositor unit 1004.
In this case, since the decoded picture data of VOP2 cannot be referred to when performing inter-VOP decoding on the VOP data of VOP3, the quality of decoded picture of VOP3 is degraded. Further, since the decoded picture data of VOP3 whose picture quality is degraded is referred to when performing inter-VOP decoding on the VOP data of VOP4, the quality of decoded picture of VOP4 is also degraded.
As described above, in the conventional video decoding apparatus, when a part of a coded stream (i.e., specific VOP data) is received with a considerable delay due to transmission path delay, the qualities of decoded picture of all VOPs, whose VOP display times are later than the VOP display time of the delayed VOP data, are degraded.
Further, in the conventional video decoding apparatus 1000, when the capacity of the buffer memory 1101 of the rearrangement unit 1001 is increased and, further, the time from when VOP data of each VOP is input as a coded stream Vin to the rearrangement unit 1001 to when the corresponding VOP data is output as a video stream VOPstr from the rearrangement unit 1001 is increased, VOP data of a VOP which arrives at the apparatus with a considerable delay can be output as a video stream VOPstr in the normal order.
In this case, however, the delay time from when the VOP data of each VOP is input as a coded stream Vin to the video decoding apparatus 1000 to when the decoded picture of each VOP is displayed is increased as a whole. As the result, especially in a monitor system in which a taken picture is transmitted to a receiving terminal and displayed in real time, the function of the monitor system is degraded by an increase in response time from when a coded stream is input to when a decoded picture is displayed.
Further, an increase in the memory capacity of the buffer memory 1101 in the rearrangement unit 1001 can lead to an increase in power consumption or device cost
The present invention is made to solve the above-described problems and has for its object to provide a video decoding method and a video decoding apparatus by which, even when a part of a coded stream obtained by coding video data is received with a considerable delay, degradation in picture quality of a decoded picture corresponding to this part of the coded stream is prevented from propagating through subsequent decoded pictures.
It is another object of the present invention to provide a program storage medium containing a program for performing the video decoding method.
Other objects and advantages of the invention will become apparent from the detailed description that follows. The detailed description and specific embodiment described are provided only for illustration since various additions and modifications within the scope of the invention will be apparent to those of skill in the art from the detailed description.
According to a first aspect of the present invention, there is provided a video decoding method for decoding coded streams obtained by coding picture data of a moving picture which is composed of pictures for which their specific times are set. This method comprises a rearrangement step of receiving coded data of the respective pictures as coded streams from a transmitting end, and outputting the received coded data of the respective pictures in chronological order of the specific times set on the respective pictures, at sending times corresponding to the respective pictures; and a decoding step of performing decoding on the coded data of the respective pictures outputted in the rearrangement step, thereby generating decoded picture data corresponding to the respective pictures. When the rearrangement step receives coded data of a picture to be outputted at a first sending time, with a delay after a second sending time which is later than the first sending time, the rearrangement step outputs coded data of pictures whose specific times are within the range from the first sending time to the second sending time. Therefore, even when coded data of a picture is received with a delay, the decoding step can resume decoding from the coded data of the delayed picture, whereby decoding on coded data can be performed without propagating degradation in picture quality through the subsequent pictures.
According to a second aspect of the present invention, in the video decoding apparatus of the first aspect, when the rearrangement step receives coded data of a picture as one of components of the moving picture, before a time at which a last picture in the display order is to be displayed, the rearrangement step outputs the coded data of the picture received. On the other hand, when the rearrangement step receives coded data of a picture as one of components of the moving picture after the time at which the last picture is to be displayed, the rearrangement step discards the coded data of the picture received. Therefore, a buffer memory for storing coded data of pictures can be effectively used, and the number of picture sequences (objects) which are simultaneously decodable is prevented from increasing unnecessarily.
According to a third aspect of the present invention, in the video decoding method of the first aspect, when the rearrangement step receives the coded data of the picture to be outputted at the first sending time, with a delay after the second sending time later than the first sending time, if it is possible to output coded data of all pictures whose specific times are within the range from the first sending time to the second sending time, the rearrangement step outputs the coded data of all these pictures. On the other hand, if it is impossible to output coded data of all pictures whose specific times are within the range from the first sending time to the second sending time, the rearrangement step discards the coded data the picture received. Therefore, inter-frame predictive decoding with referring to decoded picture data is prevented from being performed when there is no normal reference picture data, whereby degradation in picture quality is prevented from propagating through the subsequent pictures in the process of decoding.
According to a fourth aspect of the present invention, there is provided a video decoding method for decoding coded streams obtained by coding picture data of a moving picture which is composed of pictures on which their specific times are set. This method comprises a rearrangement step of receiving coded data of the respective pictures as coded streams frog a transmitting end, and outputting the received coded data of the respective pictures in chronological order of the specific times set on the respective pictures; and a decoding step of performing decoding on the coded data of the respective pictures outputted in the rearrangement process, thereby generating decoded picture data corresponding to the respective pictures. When the rearrangement step outputs coded data of a picture which is not normally decodable, the decoding step suspends decoding on coded data of pictures during a period from when the coded data is outputted to when coded data of a normally decodable picture is outputted. Therefore, considerable degradation in picture quality is prevented from occurring and, moreover, degradation in picture quality is prevented from propagating through the subsequent pictures.
According to a fifth aspect of the present invention, in the video decoding method of the fourth aspect, the decoding step decides whether or not decoding is to be suspended when coded data of a picture which is not normally decodable is outputted, on the basis of a user""s control signal. Therefore, whether or not to suspend decoding when an error is detected in coded data can be decided according to the user""s preference or purpose.
According to a sixth aspect of the present invention, in the video decoding method of the fourth aspect, the decoding step includes a shape decision step of deciding whether or not coded data of a picture outputted from the rearrangement step includes shape information indicating the shape of the picture. In the case where the coded data of the picture includes shape information, when coded data of a picture which is not normally decodable is outputted, the decoding step suspends decoding on coded data during a period from when the coded data is outputted to when coded data of a normally decodable picture is outputted. On the other hand, in the case where the coded data of the picture includes no shape information, the decoding step continues decoding on coded data regardless of whether coded data of a picture outputted from the rearrangement step is normally decodable or not. Therefore, a picture sequence having no shape signal, which is not very much affected by a transmission error or the like, can be decoded without degrading real-time performance. Further, with respect to a picture sequence having a shape signal which is considerably affected by a transmission error or the like, degradation in picture quality due to the error is reduced.
According to a seventh aspect of the present invention, there is provided a video decoding apparatus for decoding coded streams obtained by coding picture data of a moving picture which is composed of pictures on which their specific times are set. This apparatus comprises a data rearrangement unit for receiving coded data of the respective pictures as coded streams from a transmitting end, and outputting the received coded data of the pictures in chronological order of the specific times set on the respective pictures, at sending times corresponding to the respective pictures; a decoding unit for performing decoding on the coded data of the respective pictures outputted from the rearrangement unit, thereby generating decoded picture data corresponding to the respective pictures; and a display picture output unit for outputting the decoded picture data corresponding to the respective pictures outputted from the decoding unit, as display picture data, so that the respective pictures are displayed at predetermined timings. When the data rearrangement unit receives coded data of a picture to be outputted at a first sending time, with a delay after a second sending time which is later than the first sending time, the rearrangement unit outputs coded data of pictures whose specific times are within the range from the first sending time to the second sending time. Therefore, even when coded data of a picture is received with a delay, the decoding unit can resume decoding from the coded data of the delayed picture, whereby decoding on coded data can be performed without propagating degradation in picture quality.
According to an eighth aspect of the present invention, in the video decoding apparatus of Claim 7, when the data rearrangement unit outputs coded data of pictures whose specific times are within the range from the first sending time to the second sending time, the display picture output unit outputs, as display picture data, only decoded picture data of a picture whose specific time is nearest to the second sending time, among the coded data of pictures outputted from the data rearrangement unit. Therefore, the latest picture is displayed at the decoding end, among pictures which have already been decoded.
According to a ninth aspect of the present invention, there is provided a video decoding apparatus for decoding coded streams obtained by coding picture data of a moving picture which is composed of pictures on which their specific times are set. This apparatus comprises a data rearrangement unit for receiving coded data of the respective pictures as coded streams from a transmitting end, and outputting the received coded data of the respective pictures in chronological order of the specific times set on the respective pictures; a decoding unit for performing decoding on the coded data of the respective pictures outputted from the rearrangement unit, thereby generating decoded picture data corresponding to the respective pictures; and a display picture output unit for outputting the decoded picture data corresponding to the respective pictures outputted from the decoding unit, as display picture data, so that the respective pictures are displayed at predetermined timings. When coded data of a picture which is not normally decodable is outputted from the data rearrangement unit, the decoding unit suspends decoding on coded data of pictures during a period from when the coded data is outputted to when coded data of a picture which is normally decodable is outputted. Therefore, considerable degradation in picture quality is prevented from occurring and, moreover, degradation in picture quality is prevented from propagating through the subsequent pictures.
According to a tenth aspect of the present invention in the video decoding apparatus of the ninth aspect, while the decoding unit suspends decoding on coded data of pictures, the display picture output unit outputs, as display picture data, decoded picture data which has most-recently been decoded by the decoding unit. Therefore, even when decoding on coded data is being suspended, normally decoded picture is displayed.
According to an eleventh aspect of the present invention, there is provided a program storage medium containing a program for making a computer perform decoding on coded streams obtained by coding picture data of a moving picture which is composed of pictures on which their specific display times are set. This program comprises a rearrangement program for making a computer perform a rearrangement process comprising the steps of: receiving coded data of the respective pictures as coded streams from a transmitting end; outputting the received coded data of the respective pictures in chronological order of the specific times set on the respective pictures, at sending times corresponding to the respective pictures; and when coded data of a picture to be outputted at a first sending time arrives at the computer with a delay after a second sending time which is later than the first sending time, outputting coded data of pictures whose specific times are within the range from the first sending time to the second sending time; and a decoding program for making the computer perform a decoding process on the coded data of the respective pictures outputted from the rearrangement unit, thereby generating decoded picture data corresponding to the respective pictures. Therefore, even when coded data of a picture is received with a delay, the computer can resume decoding from the coded data of the delayed picture, whereby decoding on coded data can be performed without propagating degradation in picture quality through the subsequent pictures.
According to a twelfth aspect of the present invention, there is provided a program storage medium containing a program for making a computer perform decoding on coded streams obtained by coding picture data of a moving picture which is composed of pictures on which their specific times are set. This program comprises a rearrangement program for making a computer perform a rearrangement process of receiving coded data of the respective pictures as coded streams from a transmitting end, and outputting the received coded data of the respective pictures in chronological order of the specific times set on the respective pictures; and a decoding program for making the computer perform a decoding process comprising the steps of: performing decoding on the coded data of the respective pictures outputted from the rearrangement step, thereby generating decoded picture data corresponding to the respective pictures; and during the decoding, when coded data of a display picture which is not normally decodable is outputted from the rearrangement step, suspending decoding on coded data of pictures during a period from when the coded data is outputted to when coded data of a normally decodable picture is outputted. Therefore, considerable degradation in picture quality is prevented from occurring and, moreover, degradation in picture quality is prevented from propagating through the subsequent pictures.
According to a thirteenth aspect of the present invention, there is provided a video decoding method for decoding coded streams obtained by coding picture data of a moving picture which is composed of pictures for which their specific times are set. This method comprises a rearrangement step of receiving coded data of the respective pictures as coded streams from a transmitting end, and outputting the received coded data of the respective pictures in chronological order of the specific times set on the respective pictures, at sending times corresponding to the respective pictures; and a decoding step of performing decoding on the coded data of the respective pictures outputted in the rearrangement step, thereby generating decoded picture data corresponding to the respective pictures. When the rearrangement step cannot receive coded data of a picture to be outputted at a predetermined sending time by the predetermined sending time, the rearrangement step suspends sending of coded data of pictures until receiving the coded data of the picture to be outputted at the predetermined sending time, and resumes sending of coded data of pictures after receiving the coded data of the picture. Therefore, even when a decoding unit is constituted by only a decoder which performs decoding on inputted coded data, it is avoided that inter-frame prediction decoding is performed without normal reference picture data, whereby degradation in picture quality due to presence of unreceived picture is avoided. Further, degradation in picture quality, which occurs in a picture, is prevented from propagating through the subsequent pictures.
According to a fourteenth aspect of the present invention, there is provided a video decoding method for decoding coded streams obtained by coding picture data of a moving picture which is composed of pictures for which their specific times are set. This method comprises a rearrangement step of receiving coded data of the respective pictures as coded streams from a transmitting end, and outputting the received coded data of the respective pictures in chronological order of the specific times set on the respective pictures, at sending times corresponding to the respective pictures; and a decoding step of performing decoding on the coded data of the respective pictures outputted in the rearrangement step, thereby generating decoded picture data corresponding to the respective pictures. When the rearrangement step cannot receive coded data of a picture to be outputted at a predetermined sending time by the predetermined sending time, the rearrangement step suspends sending of coded data of pictures until a predetermined condition is satisfied or until it receives the coded data of the picture to be outputted at the predetermined sending time, and resumes sending of coded data of pictures after the condition is satisfied or after it receives the coded data of the picture. Therefore, degradation in picture quality due to presence of coded data of an unreceived picture is avoided and, moreover, the operation of a buffer memory is prevented from being impeded by unnecessary coded data.
According to a fifteenth aspect of the present invention, in the video decoding method of the fourteenth aspect, the predetermined condition is that, in the rearrangement step, the remaining memory capacity of a buffer memory which stores received coded streams of the respective pictures is equal to or smaller than a predetermined threshold in the rearrangement step. Therefore, overflow of the buffer memory is avoided.
According to a sixteenth aspect of the present invention, in the video decoding method of the fourteenth aspect, the predetermined condition is that, in the rearrangement step, the time during which sending of coded data of pictures is suspended is equal to or longer than a predetermined threshold. Therefore, when part of coded data of pictures from a delayed picture to a target picture to be displayed at this point of time is deleted from the buffer memory and so these coded data cannot be decoded normally, the coded data of the delayed picture is prevented from being output to the buffer memory.
According to a seventeenth aspect of the present invention, there is provided a video decoding apparatus for decoding coded streams obtained by coding picture data of a moving picture which is composed of pictures on which their specific times are set. This apparatus comprises a data rearrangement unit for receiving coded data of the respective pictures as coded streams from a transmitting end, and outputting the receive coded data of the pictures in chronological order of the specific times set on the respective pictures, at sending times corresponding to the respective pictures; a decoding unit for performing decoding on the coded data of the respective pictures outputted from the rearrangement unit, thereby generating decoded picture data corresponding to the respective pictures; and a display picture output unit for outputting the decoded picture data corresponding to the respective pictures outputted from the decoding unit, as display picture data, so that the respective pictures are displayed at predetermined timings. When the data rearrangement unit cannot receive coded data of a picture to be outputted at a predetermined sending time by the predetermined sending time, the rearrangement unit suspends sending of coded data of pictures until receiving the coded data of the picture to be outputted at the predetermined sending time, and resumes sending of coded data of pictures after receiving the coded data of the picture. Therefore, even when the decoding unit is constituted by only a decoder which performs decoding on inputted coded data, it is avoided that inter-frame prediction decoding is performed without normal reference picture data, whereby degradation in picture quality due to presence of unreceived picture is avoided. Further, degradation in picture quality, which occurs in a picture, is prevented from propagating through the subsequent pictures.
According to an eighteenth aspect of the present invention, there is provided a video decoding apparatus for decoding coded streams obtained by coding picture data of a moving picture which is composed of pictures on which their specific times are set. This apparatus comprises a data rearrangement unit for receiving coded data of the respective pictures as coded streams from a transmitting end, and outputting the received coded data of the pictures in chronological order of the specific times set on the respective pictures, at sending times corresponding to the respective pictures; a decoding unit for performing decoding on the coded data of the respective pictures outputted from the rearrangement unit, thereby generating decoded picture data corresponding to the respective pictures; and a display picture output unit for outputting the decoded picture data corresponding to the respective pictures outputted from the decoding unit, as display picture data, so that the respective pictures are displayed at predetermined timings. When the data rearrangement unit cannot receive coded data of a picture to be outputted at a predetermined sending time by the predetermined sending time, the rearrangement unit suspends sending of coded data of pictures until a predetermined condition is satisfied or until it receives the coded data of the picture to be outputted at the predetermined sending time, and resumes sending of coded data of pictures after the condition is satisfied or after it receives the coded data of the picture. Therefore, degradation in picture quality due to presence of coded data of an unreceived picture is avoided and, moreover, the operation of a buffer memory is prevented from being impeded by unnecessary coded data.
According to a nineteenth aspect of the present invention, in the video decoding apparatus of Claim the eighteenth aspect, when the data rearrangement unit resumes sending of coded data of pictures to the decoding unit, the display image output unit performs display of pictures within a predetermined period of time such that delays in display timings of these pictures, which occur due to the suspended sending, are decreased every time a picture is displayed. Therefore, when coded data of a picture is received with a delay, a predetermined number of pictures following this delayed picture are displayed such that the actual display times of these pictures, which are delayed from their original display times, gradually approach the original display times. Therefore, even when there is such delayed picture, the display times of the subsequent pictures are gradually corrected, resulting in a smoothly-moving picture.
According to a twentieth aspect of the present invention, there is provided a program storage medium containing a program for making a computer perform decoding on coded streams obtained by coding picture data of a moving picture which is composed of pictures on which their specific display times are set. This program comprises a rearrangement program for making a computer perform a rearrangement process comprising the steps of: receiving coded data of the respective pictures as coded streams from a transmitting end; outputting the received coded data of the respective pictures in chronological order of the specific times set on the respective pictures, at sending times corresponding to the respective pictures; and when coded data of a picture to be outputted at a predetermined sending time has not arrived at the computer by the predetermined sending time, suspending sending of coded data of pictures until receiving the coded data of the picture to be outputted at the predetermined sending time, and resuming sending of coded data of pictures after receiving the coded data of the picture; and a decoding program for making the, computer perform a decoding process on the coded data of the respective pictures outputted from the rearrangement unit, thereby generating decoded picture data corresponding to the respective pictures. Therefore, even when a decoding unit is constituted by only a decoder which performs decoding on inputted coded data, it is avoided that inter-frame prediction decoding is performed without normal reference picture data, whereby degradation in picture quality due to presence of unreceived picture is avoided. Further, degradation in picture quality which occurs in a picture is prevented from propagating through the subsequent pictures.
According to a twenty-first aspect of the present invention, there is provided a program storage medium containing a program for making a computer perform decoding on coded streams obtained by coding picture data of a moving picture which is composed of pictures on which their specific display times are set. This program comprises a rearrangement program for making a computer perform a rearrangement process comprising the steps of: receiving coded data of the respective pictures as coded streams from a transmitting end; outputting the received coded data of the respective pictures in chronological order of the specific times set on the respective pictures, at sending times corresponding to the respective pictures; and when coded data of a picture to be outputted at a predetermined sending time has not arrived at the computer by the predetermined sending time, suspending sending of coded data of pictures until a predetermined condition is satisfied or until the computer receives the coded data of the picture to be outputted at the predetermined sending time, and resuming sending of coded data of pictures after the condition is satisfied or after the computer receives the coded data of the picture; and a decoding program for making the computer perform a decoding process on the coded data of the respective pictures outputted from the rearrangement unit, thereby generating decoded picture data corresponding to the respective pictures. Therefore, degradation in picture quality due to presence of coded data of unreceived picture is avoided and, moreover, the operation of a buffer memory is prevented from being impeded by unnecessary coded data.